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CIRCUITRY FOR DETERMINING MEDIAN 
OF IMAGE PORTIONS 


CROSS REFERENCE TO RELATED 

APPLICATIONS 5 

This application claims the benefit of the U.S. Provisional 
Application No. 60/052,399, filed on Jul. 4, 1997, which is 
hereby incorporated by reference, and this application is a 
Continuation, under 35 USC 120, of U.S. Utility application 
Ser. No. 09/115,318, filed on Jul. 14, 1998, now U.S. Pat. 
No. 6,546,148, incorporated by reference. 


STATEMENT AS TO FEDERALLY SPONSORED 
RESEARCH 
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The invention described herein was made in the perfor- 
mance of work under a NASA contract, and is subject to the 
provisions of Public Law 96-517 (35 U.S.C. 202) in which 
the Contractor has elected to retain title. 


FIELD OF THE INVENTION 
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The present invention teaches enhancing fabrication yield 
of image sensors by using circuitry which compensates for 
faulty image sensor areas. ^ 


BACKGROUND 


Consumer requirements have required larger and larger 
sensor arrays. However, as a sensor array gets larger, it 
becomes more difficult to reliably fabricate the sensor array. 30 
Fabrication yield and sensor cost can be proportional to the 
size of the sensor array. 

Typically, any bad portion in a sensor array requires that 
the sensor array be discarded or used for another purpose. 
However, there have been attempts to incorporate circuitry 35 
which compensates for fabrication errors. 

Such errors can be corrected using a median of a neigh- 
borhood surrounding the faulty pixel. 

The median also has other applications which are well 4Q 
known in the art. 


SUMMARY 

The present disclosure describes performing the median 
function over a pixel neighborhood. The result of that 45 
median function can be used to simulate an effect which is 
perceived by the user. According to the present system, 
on-chip processing is used to hardware -calculate the median 
function. 

A number of different embodiments are described to use 50 
hardware to calculate the median function. That hardware is 
preferably on the same chip as the image sensor. Most 
preferably, this uses active pixel sensors, in which some 
circuitry is integrated onto the substrate along with the 
photosensor. 55 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other aspects of the invention will now be 
described with reference to the attached drawings, in which: 

FIG. 1 shows a prior art hardware technique of finding the 
median value; 

FIG. 2 shows an embodiment of a median calculating 
circuit for an active pixel sensor; 

FIG. 3 shows a block diagram of the median calculation 65 
operations as carried out with a buffer in an active pixel 
array; 


FIG. 4 shows an alternative embodiment in which the 
median value itself is outputted instead of the position of the 
median value; 

FIG. 5 shows a block diagram of the FIG. 4 alternative 
embodiment; 

FIG. 6 shows an alternative embodiment which selec- 
tively adds and removes items to find the median; 

FIG. 7 shows an embodiment using winner-take- all cir- 
cuitry. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

A median filter operates on a neighborhood of pixels. The 
most commonly-used sizes include 3x3 and 5x5 neighbor- 
hood sizes. The median function seeks to determine, among 
the neighborhood of NxN pixels, which pixel among those 
NxN pixels has the median amount of brightness. The 
median is always exactly one of those pixels, and represents 
the one pixel which represents the best fit. The present 
system proposes using a 3x3 neighborhood as the best 
tradeoff in an active pixel sensor. 

Different ways are known in the art for forming a median 
value from the 3x3 square neighborhood. This involves 
different rows and columns, requiring obtaining physically 
separated values. One interesting way of forming a median 
is described in the article “A Two Dimensional Visual 
Tracking Array” (Steven P. DeWeerth and Carver A. Mead, 
MIT Press 1988). This article describes using hardware in a 
VLSI chip to track the center intensity of a visual field. One 
application of this circuit is in computing the median value. 

The system as disclosed uses a network of resistors to 
form a ladder, n resistors are connected in series, with 
voltages V 0 and W n applied to the ends of the resistive line. 
This produces a linear voltage gradient along the array, with 
the voltage at the Nth node given by: 


A transconductance amplifier is formed of a differential 
stage that produces an output current as a function of the bias 
current, and the differential input voltage. The output current 
corresponds to the bias current multiplied by the difference 
between the input voltages. 

In the system described in this article, the transconduc- 
tance amplifier is replaced with a phototransistor of a 
light-receiving element. The phototransistor produces a cur- 
rent that is proportional to light intensity. This current is used 
as the bias current. The input voltages to the amplifiers are 
obtained from the resistive line. Hence, the output of each 
amplifier corresponds to the input of the amplifier weighted 
by the corresponding light intensity. That output is fed back 
to the input as the differential current. A higher bias input 
produces more output, and hence makes the output of the 
amplifier more closely resemble the input. The amplifier 
with the highest output, therefore, represents the position 
whose value received the highest input. A two-dimensional 
array can be used to find the median of a 3x3 system. The 
two-dimensional array (FIG. 7 of this article) is shown in 
FIG. 1. Each receptor 100 is alternated so that the currents 
from adjacent receptors are added to opposing axes. 

While the above has described how such a system would 
operate, it should be understood that any similar median 
obtaining structure could be used. 

The original circuit as proposed included a differential 
pair within each pixel. The bias voltage was provided by a 
photoreceptor. 
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However, this system had drawbacks. The fill factor of 
such a system is relatively low. In addition, this system 
actually did not provide the pixel readout value: it only 
provided the median value. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

The present embodiment uses this or similar median 
calculating circuitry to form the median of a desired neigh- 
borhood of pixels within an active pixel array circuit. An 
embodiment is shown in FIG. 2. Atypical active pixel sensor 
array 202 is of the general type described in U.S. Pat. No. 
5,471,515, the disclosure of which is incorporated by ref- 
erence herein. 

The actual image sensor array 200 is connected to a row 
decoder 202 and a column decoder 204. These two elements 
provide input to the active pixel sensor array which produces 
outputs 210. The outputs can be produced individually, or in 
some parallel form such as column parallel. The system 
shown in FIG. 2 is preferably column, parallel. This system 
needs a buffer of three rows for each 3x3 element median 
calculation. Each new row is added to the active part of the 
buffer 220 at each time, and the oldest row of the three is 
dropped off at that point. 

When the buffer is properly loaded with the three values 
therein, the median calculating circuit 250 system operates 
on adjacent neighborhoods of 3x3 at each time. Each 3x3 
operation allows providing the median results of all the 
pixels in a certain portion. Therefore, in the first cycle, the 
group 222 is medianed, thus providing the median of the first 
value (col. 1, row 1), 4th (col. 2, row 1), and 7th (col. 3, row 

1) values. Similarly, in the second cycle, the 2nd (col. 1, row 

2) , 5th and 8th values are calculated, and in the third cycle, 
the 3rd, 6th and 9th values are calculated. This provides the 
median at the time X. 

At the next time period, everything is shifted over by one 
column by changing the control values. This reconfigures 
the connections to the buffer 220. These connections are 
stored for example in shift register 230. Buffer parts 226 and 
228 keep their values (columns 2 and 3 respectively). Buffer 
part 224 is filled with the contents of column 4. Similarly, at 
time (X+2), buffer 226 contains the oldest information and 
is dropped and replaced with column 5’s contents. The 
operation can be done in circuitry by using, for example, 
shift register 230. Alternatively, these reconfigurations can 
be handled by a microcontroller or ASIC. 

FIG. 3 shows using a 3 row buffer in a column-parallel 
implementation. This shows the group 300 used for the X 
median calculation as well as the group used for the X+l 
median calculation 302 and for the X+3 median calculation 
304. The subsequent median calculations are carried out by 
reconfiguring the neighborhood that is presently being cal- 
culated. 

Other control values are also stored in shift register 230 to 
effect the pixel input and output connection. These control 
values include the values for the resistive ladder which can 
include horizontal and vertical resistor valves RH and RV, 
beginning and end voltages, Vo and Vn. These values can be 
changed in sequence to remake the proper connection. The 
connections are made by on and off transistors such as 
241-249. Each column can include such transistors 242, 
244, 246. Turning on a transistor connects that column to the 
buffer. The resistor ladder values are controlled by transis- 
tors such as 248, 249 which pass their values in the linear 
mode. 

The shift register 230 also stores compensation values for 
operating at the top and bottom part of the array. This uses 
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an end of frame signal as the last row read out. The end of 
frame signal causes that row to be read into the buffer two 
or more times in order to obtain a more appropriate median 
result. The operation is controlled by logic circuit 260, 
5 which can include a microcontroller or logic values. 

The same problem occurs also with the beginning and the 
end of the row. The same solution could be executed by logic 
circuit 260, i.e. padding of the first and last column pixels. 

An alternative is to output only the “real” median values. 
10 In this case, the effective array size can be smaller than the 
original, by four pixels (two at the beginning and two at the 
end) in each dimension. This alternative is also programmed 
into logic circuit 260. 

5 An alternative embodiment shown in FIG. 4 operates 
without an encoder operation, i.e. without resistive ladder 
252. All differential pairs are biased by the same specific 
current 400. The sensor’s voltages such as 402 are input 
directly to one of the inputs of the differential amplifier. The 
2Q other input is connected, like in the FIG. 1 circuit, to the 
common amplifier output. This embodiment, instead of 
outputting an encoded value indicative of the median, actu- 
ally outputs the value of the median at 410. This latter 
system enables using simpler switching, since switches for 
25 position encoding, are not necessary. 

Only switches such as 249, 500, 502, 504 for connecting 
the 3x3 differential pairs to a common load and amplifier are 
necessary as shown in FIG. 5. 

Each column of 3 could be looked upon as one basic 
30 element that needs to be connected/disconnected to a certain 
amplifier as shown in FIG. 5. As before, processing of every 
3 contiguous columns is done in parallel. FIG. 4 shows one 
of those units. Three cycles are required to provide the 
median results for the whole row. 

35 Another embodiment is shown in FIG. 6. A neighborhood 
of values is read from buffer 220 through switches 602, 604, 
606, to source followers 622, 624, 626 to a common line 608 
shown with its stray capacitance. While only three connec- 
tions are shown, it should be understood that many more 
40 connections are possible. The operation is controlled by 
controller 610. The line is initially charged. After the initial 
charging, at each clock cycle, one of the inputs is discon- 
nected by control element 610 which controls gates at the 
source followers. 

The output 608 is connected to transconductance ampli- 
fiers of the previously-discussed type 612. The disconnected 
value which changes the result the most is the highest result. 
This value is inhibited by control circuit 610. 

5Q Then, the same procedure is repeated on the remaining 
input voltages. The value whose disconnection affects the 
output the most is again highest of the inputs, making this 
the second highest value. This procedure is repeated until the 
median is found. If a neighborhood of 9 is used, the 
55 operation requires 9+8+7+6+5 cycles to find the median. A 
3 row buffer is still required, and switches for passing the 
appropriate values to the source follower is also required. 
FIG. 6 shows the controller detecting which value causes the 
most change based on outputs of the transconductance 
60 amplifier. 

Another embodiment uses winner- take -all (“WTA”) cir- 
cuitry 700 with the APS array. Different WTA circuits, such 
as those shown in U.S. Pat. Nos. 5,561,287 and 5,059,814 
are known. 

65 Each winner is inhibited for the truncated integer (N/2) 
where N is the total number of elements. Here, that is 4 times 
for a 3x3=9 neighborhood. The next winner would be the 
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median. This will require 5 cycles of operation. This circuit 
can also be used for finding the runner-up, etc. 

If speed is the main consideration, additional circuitry can 
enable the median output in one cycle. Using a modified 
WTA circuit where K winners are selected at a time as 
shown in FIG. 7 enables the median operation in two cycles. 

For the 3x3 area, the 4 winners at first using a 4-WTA circuit 
and then the median at the second cycle, using a 1-WTA. 

A3 row buffer 220 is still required, and switches 702 for 
passing the appropriate values to the WTA. 

Although only a few embodiments have been described in 
detail above, other embodiments are contemplated by the 
inventor and predictable based on the disclosure. All of these 
embodiments are intended to be encompassed within the 
following claims. In addition, other modifications are con- 
templated and are also intended to be covered. 

What is claimed: 

1. An image sensor on a chip, the image sensor compris- 
ing: 

a pixel sensor array having column parallel pixel read-out; 

a pixel-value analog-data storage buffer coupled with the 
pixel sensor array to receive analog-data from the 
column parallel pixel read-out, the pixel-value analog- 
data storage buffer comprising at least three rows of 
storage for pixel-value analog data; 

median-obtaining circuitry; and 

switching circuitry coupled with the pixel-value analog- 
data storage buffer and with the median-obtaining 
circuitry to connect different neighborhoods of the 
pixel-value analog-data storage buffer with the median- 
obtaining circuitry during at least three processing 
cycles. 

2. The image sensor of claim 1, wherein the different 
neighborhoods each comprise a three-by- three pixel-data 
neighborhood in the pixel-value analog-data storage buffer. 

3. The image sensor of claim 1, further comprising: 

an end-of-frame signal line coupled with the column 

parallel pixel read-out to cause the pixel- value analog- 
data storage buffer to receive sequential same -row 
analog data. 

4. The image sensor of claim 3, wherein the pixel-value 
analog-data storage buffer is wider than the pixel sensor 
array. 

5. The image sensor of claim 1, wherein the median- 
obtaining circuitry comprises a plurality of differential pairs. 

6. The image sensor of claim 5, wherein the pixel-value 
analog-data storage buffer comprises a plurality of capaci- 
tors. 

7. The image sensor of claim 6, wherein the switching 
circuitry connects voltages from the plurality of capacitors 5Q 
to inputs of the plurality of differential pairs. 

8. The image sensor of claim 6, wherein the median- 
obtaining circuitry further comprises a resistive divider, and 
wherein the switching circuitry connects voltages from the 
resistive divider to inputs of the plurality of differential 
pairs. 
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9. The image sensor of claim 1, wherein the median- 
obtaining circuitry comprises maximum-obtaining circuitry. 

10. The image sensor of claim 9, wherein the maximum- 
obtaining circuitry comprises source followers tied to a 

5 common output line. 

11. The image sensor of claim 9, wherein the maximum- 
obtaining circuitry comprises winner-take-all circuitry. 

12. A method of fabricating an image sensor on a semi- 
conductor chip, the method comprising: 

forming a pixel sensor array on the semiconductor chip; 
forming a pixel- value analog-data storage buffer compris- 
ing at least three rows on the semiconductor chip; 
forming median-obtaining circuitry on the semiconductor 
chip and coupled with the pixel-value analog-data 
storage buffer, the median-obtaining circuitry including 
switching circuitry to connect different neighborhoods 
of the pixel- value analog-data storage buffer with the 
median-obtaining circuitry during at least three pro- 
cessing cycles; and 

coupling the pixel-value analog-data storage buffer with 
the pixel sensor array on the semiconductor chip such 
that the pixel-value analog-data storage buffer can 
receive analog data from the pixel sensor array when in 
operation. 

13. The method of claim 12, wherein the different neigh- 
borhoods each comprise a five-by-five pixel-data neighbor- 
hood in the pixel-value analog-data storage buffer. 

14. The method of claim 13, further comprising: 
forming an end-of-frame signal line on the semiconductor 

chip such that the pixel-value analog-data storage 
buffer can receive sequential same-row analog data 
from the pixel sensor array when in operation. 

15. The method of claim 12, wherein the pixel sensor 
array has an actual array size in two dimensions, and 
wherein the median-obtaining circuitry outputs an effective 
array size smaller than the actual array size for the pixel 
sensor array in each of the two dimensions. 

16. The method of claim 15, wherein the median- 
obtaining circuitry comprises a plurality of differential pairs. 

17. The method of claim 16, wherein the pixel-value 
analog-data storage buffer comprises a plurality of capaci- 
tors. 

18. The method of claim 17, wherein the switching 
circuitry connects voltages from the plurality of capacitors 
to inputs of the plurality of differential pairs. 

19. The method of claim 15, wherein the median- 
obtaining circuitry comprises maximum-obtaining circuitry. 

20. The method of claim 19, wherein the maximum- 
obtaining circuitry comprises source followers tied to a 
common output line. 

21. The method of claim 19, wherein the maximum- 
obtaining circuitry comprises winner-take-all circuitry. 





